Streaming Action Fulfillment Based on Partial Hypotheses

ABSTRACT

A method for streaming action fulfillment receives audio data corresponding to an utterance where the utterance includes a query to perform an action that requires performance of a sequence of sub-actions in order to fulfill the action. While receiving the audio data, but before receiving an end of speech condition, the method processes the audio data to generate intermediate automated speech recognition (ASR) results, performs partial query interpretation on the intermediate ASR results to determine whether the intermediate ASR results identify an application type needed to perform the action and, when the intermediate ASR results identify a particular application type, performs a first sub-action in the sequence of sub-actions by launching a first application to execute on the user device where the first application is associated with the particular application type. The method, in response to receiving an end of speech condition, fulfills performance of the action.

CROSS-REFERENCE TO RELATED APPLICATIONS

This U.S. patent application is a continuation of, and claims priorityunder 35 U.S.C. § 120 from, U.S. patent application Ser. No. 17/247,334,filed on Dec. 8, 2020. The disclosure of this prior application isconsidered part of the disclosure of this application and is herebyincorporated by reference in its entirety

TECHNICAL FIELD

This disclosure relates to streaming action fulfillment based on partialhypotheses

BACKGROUND

Users frequently interact with voice-enabled devices, such as smartphones, smart watches, and smart speakers, through digital assistantinterfaces. These digital assistant interfaces enable users to completetasks and obtain answers to questions they have all through natural,conversational interactions. Ideally, when conversing with a digitalassistant interface, a user should be able to communicate as if the userwere talking to another person, via spoken requests directed towardtheir voice-enabled device running the digital assistant interface. Thedigital assistant interface will provide these spoken requests to anautomated speech recognizer to process and recognize the spoken requestso that an action can be performed.

Digital assistant interfaces are moving onto mobile devices in whichmuch of the speech recognition processing occurs on-device withoutincurring the added latency to connect to a server via a network andsend audio data to the server to perform speech recognition usingcloud-based speech recognition servers. In addition to improvinglatency, other benefits of on-device speech recognition include improvedreliability and privacy. As such, digital assistant interfaces arebecoming deeply integrated with various applications and operatingsystems running on mobile devices, thereby enabling a user to controltheir mobile device solely using their voice. However, theseapplications installed on user devices may themselves be slow,unreliable, or require network access to servers, thereby throttling thebenefits that the on-device processing capabilities the digitalassistant interface affords and can lead to a sluggish user experience.

SUMMARY

One aspect of the disclosure provides a method for streaming actionfulfillment. The method includes receiving, at data processing hardware,audio data corresponding to an utterance spoken by a user of a userdevice where the utterance includes a query to perform an action wherethe query requires performance of a sequence of sub-actions in order tofulfill the action. While receiving the audio data, but before receivingan end of speech condition, the method also includes processing, by thedata processing hardware, using a speech recognizer, a first portion ofthe received audio data to generate a first sequence of intermediateautomated speech recognition (ASR) results. While receiving the audiodata, but before receiving an end of speech condition, the methodfurther includes performing, by the data processing hardware, partialquery interpretation on the first sequence of intermediate ASR resultsto determine whether the first sequence of intermediate ASR resultsidentifies an application type needed to perform the action and, whenthe first sequence of intermediate ASR results identifies a particularapplication type, performing, by the data processing hardware, a firstsub-action in the sequence of sub-actions by launching a firstapplication to execute on the user device where the first application isassociated with the particular application type. The method additionallyincludes, in response to receiving an end of speech condition,fulfilling, by the data processing hardware, performance of the action.Determining the end of speech condition may include detecting, using thespeech recognizer, at least a minimum duration of non-speech in thereceived audio.

Implementations of the disclosure may include one or more of thefollowing optional features. In some implementations, while receivingthe audio data before receiving the end of speech condition and afterlaunching the first application, the method also includes processing, bythe data processing hardware, using the speech recognizer, a secondportion of the received audio data to generate a second sequence ofintermediate ASR results, performing, by the data processing hardware,the partial query interpretation on the second sequence of intermediateASR results to identify a search query for content in the firstapplication, and performing, by the data processing hardware, a secondsub-action in the sequence of actions by instructing the firstapplication to perform the search query for the content in the firstapplication. In these implementations, in response to launching thefirst application to execute on the user device, the method alsoincludes displaying, by the data processing hardware, in a graphicaluser interface of the user device, an initial screen having a searchfield and/or multiple graphical elements each representing differentcategories of content in the first application. Here, instructing thefirst application to perform the search query for the content includesentering, into the search field of the first screen, text characterizingthe search query for the content in the first application or selecting,from among the multiple graphical elements of the first screen, thegraphical element representing the category of content that includes thecontent specified by the search query. These implementations also mayinclude, after the first application performs the search query for thecontent, displaying, by the data processing hardware, in the graphicaluser interface, a second screen presenting one or more results thatinclude the content specified by the search query. In theseimplementations, the method may further include receiving, at the dataprocessing hardware, a user input indication indicating selection of oneof the results presented in the second screen and, in response toreceiving the user input indication, detecting, by the data processinghardware, the end of speech condition. These implementations may alsoinclude, after the first application performs the search query for thecontent, processing, by the data processing hardware, using the speechrecognizer, a third portion of the received audio data to generate athird sequence of intermediate ASR results, performing, by the dataprocessing hardware, the partial query interpretation on the thirdsequence of intermediate ASR results to determine whether to update thesearch query for more specific content, and, when the partial queryinterpretation performed on the third sequence of intermediate searchresults determines to update the search query for more specific content,performing, by the data processing hardware, a third sub-action in thesequence of sub-actions by instructing the first application to updatethe search query for the more specific content.

In some configurations, performing the partial query interpretation onthe first sequence of intermediate ASR results may determine that thefirst sequence of intermediate ASR results identifies the particularapplication type and fails to specify a slot value associated withnaming a specific application for use in fulfilling the action. Here,performing the first sub-action in the sequence of sub-actions includeslaunching the first application as a default application associated withthe particular application type. In these configurations, whilereceiving the audio data before receiving the end of speech conditionand after launching the first application, the method also includesprocessing, by the data processing hardware, using the speechrecognizer, a second portion of the received audio data to generate asecond sequence of intermediate ASR results and performing, by the dataprocessing hardware, the partial query interpretation on the secondsequence of intermediate ASR results to determine whether the secondsequence of intermediate ASR results identify a second application touse for fulfilling the action. Additionally, in these configurations,when the second sequence of intermediate ASR results identify the secondapplication for use in fulfilling the action, the method furtherincludes rolling-back, by the data processing hardware, performance ofthe first sub-action by ceasing execution of the first application onthe user device and re-performing, by the data processing hardware, thefirst sub-action in the sequence of actions by launching the secondapplication to execute on the user device.

In some examples, while receiving the audio data before receiving theend of speech condition and after launching the first application, themethod further includes processing, by the data processing hardware,using the speech recognizer, a second portion of the received audio datato generate a second sequence of intermediate ASR results andperforming, by the data processing hardware, the partial queryinterpretation on the second sequence of intermediate ASR results toidentify a second sub-action in the sequence of sub-actions. In theseexamples, while receiving the audio data before receiving the end ofspeech condition and after launching the first application, the methodalso includes determining, by the data processing hardware, a rollbackfeasibility score associated with the second sub-action where therollback feasibility indicates a likelihood that a user experience willbe degraded if executing the second sub-action has to be rolled backand, when the rollback feasibility score satisfies a rollbackfeasibility threshold, performing, by the data processing hardware, thesecond sub-action. When the rollback feasibility score does not satisfythe rollback feasibility threshold, the method may delay performing, bythe data processing hardware, the second sub-action until the end ofspeech condition is received. Also in these examples, the method mayfurther include determining, by the data processing hardware, aconfidence score of second sub-action identified by performing thepartial query interpretation on the second sequence of intermediate ASRresults and when the confidence score of the second sub-action fails tosatisfy a confidence threshold, prompting, by the data processinghardware, the user to confirm whether the second sub-action is correctlyidentified.

Another aspect of the disclosure provides a system for streaming actionfulfillment. The system includes data processing hardware and memoryhardware in communication with the data processing hardware. The memoryhardware stores instructions that when executed on the data processinghardware cause the data processing hardware to perform operations. Theoperations include receiving audio data corresponding to an utterancespoken by a user of a user device where the utterance includes a queryto perform an action where the query requires performance of a sequenceof sub-actions in order to fulfill the action. While receiving the audiodata, but before receiving an end of speech condition, the operationsalso include processing, using a speech recognizer, a first portion ofthe received audio data to generate a first sequence of intermediateautomated speech recognition (ASR) results. While receiving the audiodata, but before receiving an end of speech condition, the operationsfurther include performing partial query interpretation on the firstsequence of intermediate ASR results to determine whether the firstsequence of intermediate ASR results identifies an application typeneeded to perform the action and, when the first sequence ofintermediate ASR results identifies a particular application type,performing a first sub-action in the sequence of sub-actions bylaunching a first application to execute on the user device where thefirst application is associated with the particular application type.The operations additionally includes, in response to receiving an end ofspeech condition, fulfilling performance of the action. Determining theend of speech condition may include detecting, using the speechrecognizer, at least a minimum duration of non-speech in the receivedaudio.

Implementations of the disclosure may include one or more of thefollowing optional features. In some implementations, while receivingthe audio data before receiving the end of speech condition and afterlaunching the first application, the operations also include processing,using the speech recognizer, a second portion of the received audio datato generate a second sequence of intermediate ASR results, performingthe partial query interpretation on the second sequence of intermediateASR results to identify a search query for content in the firstapplication, and performing a second sub-action in the sequence ofactions by instructing the first application to perform the search queryfor the content in the first application. In these implementations, inresponse to launching the first application to execute on the userdevice, the operations also include displaying in a graphical userinterface of the user device, an initial screen having a search fieldand/or multiple graphical elements each representing differentcategories of content in the first application. Here, instructing thefirst application to perform the search query for the content includesentering, into the search field of the first screen, text characterizingthe search query for the content in the first application or selecting,from among the multiple graphical elements of the first screen, thegraphical element representing the category of content that includes thecontent specified by the search query. These implementations also mayinclude, after the first application performs the search query for thecontent, displaying in the graphical user interface, a second screenpresenting one or more results that include the content specified by thesearch query. In these implementations, the operations may furtherinclude receiving a user input indication indicating selection of one ofthe results presented in the second screen and, in response to receivingthe user input indication, detecting the end of speech condition. Theseimplementations may also include, after the first application performsthe search query for the content, processing using the speechrecognizer, a third portion of the received audio data to generate athird sequence of intermediate ASR results, performing the partial queryinterpretation on the third sequence of intermediate ASR results todetermine whether to update the search query for more specific content,and, when the partial query interpretation performed on the thirdsequence of intermediate search results determines to update the searchquery for more specific content, performing a third sub-action in thesequence of sub-actions by instructing the first application to updatethe search query for the more specific content.

In some configurations, performing the partial query interpretation onthe first sequence of intermediate ASR results may determine that thefirst sequence of intermediate ASR results identifies the particularapplication type and fails to specify a slot value associated withnaming a specific application for use in fulfilling the action. Here,performing the first sub-action in the sequence of sub-actions includeslaunching the first application as a default application associated withthe particular application type. In these configurations, whilereceiving the audio data before receiving the end of speech conditionand after launching the first application, the operations also includeprocessing, using the speech recognizer, a second portion of thereceived audio data to generate a second sequence of intermediate ASRresults and performing, by the data processing hardware, the partialquery interpretation on the second sequence of intermediate ASR resultsto determine whether the second sequence of intermediate ASR resultsidentify a second application to use for fulfilling the action.Additionally, in these configurations, when the second sequence ofintermediate ASR results identify the second application for use infulfilling the action, the operations further include rolling-backperformance of the first sub-action by ceasing execution of the firstapplication on the user device and re-performing, by the data processinghardware, the first sub-action in the sequence of actions by launchingthe second application to execute on the user device.

In some examples, while receiving the audio data before receiving theend of speech condition and after launching the first application, theoperations further include processing, using the speech recognizer, asecond portion of the received audio data to generate a second sequenceof intermediate ASR results and performing, by the data processinghardware, the partial query interpretation on the second sequence ofintermediate ASR results to identify a second sub-action in the sequenceof sub-actions. In these examples, while receiving the audio data beforereceiving the end of speech condition and after launching the firstapplication, the operations also include determining a rollbackfeasibility score associated with the second sub-action where therollback feasibility indicates a likelihood that a user experience willbe degraded if executing the second sub-action has to be rolled backand, when the rollback feasibility score satisfies a rollbackfeasibility threshold, performing, by the data processing hardware, thesecond sub-action. When the rollback feasibility score does not satisfythe rollback feasibility threshold, the operations may delay performingthe second sub-action until the end of speech condition is received.Also in these examples, the operations may further include determining,by the data processing hardware, a confidence score of second sub-actionidentified by performing the partial query interpretation on the secondsequence of intermediate ASR results and when the confidence score ofthe second sub-action fails to satisfy a confidence threshold,prompting, by the data processing hardware, the user to confirm whetherthe second sub-action is correctly identified.

The details of one or more implementations of the disclosure are setforth in the accompanying drawings and the description below. Otheraspects, features, and advantages will be apparent from the descriptionand drawings, and from the claims.

DESCRIPTION OF DRAWINGS

FIG. 1 is a schematic view of an example speech environment forperforming fulfillment based on partial speech recognition results.

FIG. 2A-2H are schematic views of example assistant interfaces for thespeech environment of FIG. 1 .

FIG. 3 is a flowchart of an example arrangement of operations for amethod of performing fulfillment based on partial speech recognitionresults.

FIG. 4 is a schematic view of an example computing device that may beused to implement the systems and methods described herein.

Like reference symbols in the various drawings indicate like elements.

DETAILED DESCRIPTION

Conventionally, digital assistant interfaces that enable users tocomplete tasks and obtain answers to questions they have throughnatural, conversational interactions have required the processing ofservers to support speech recognition and language understanding models.These server-based models were not suitable for execution on device dueto their size, which could exceed the available storage or memory on thedevice. However, recent advancements in recurrent neural networks haveenabled the development of new speech recognition and languageunderstanding models of drastically reduced size (e.g., less than half agigabyte) suitable for storage and processing on-device. As such,digital assistant interfaces are moving onto mobile devices in whichmuch of the speech recognition processing occurs on-device withoutincurring the added latency to connect to a server via a network andsend audio data to the server to perform speech recognition usingcloud-based speech recognition servers. User experience is drasticallyenhanced since transcriptions of speech can occur in real-time andwithout a network connection. In addition to improving latency, otherbenefits of on-device speech recognition include improved reliabilityand privacy. As such, digital assistant interfaces are becoming deeplyintegrated with various applications and operating systems running onmobile devices, thereby enabling a user to control their mobile devicesolely using their voice. For example, a user could multi-task acrossmultiple applications running on the device, such as creating a calendarinvite, finding and sharing a photo with friends, or dictating an email.However, these applications installed on user devices may themselves beslow, unreliable, or require network access to servers, therebythrottling the benefits that the on-device digital assistant interfaceaffords and can lead to a sluggish user experience.

Generally, in speech recognition systems, speech endpointing is theprocess of determining which part of incoming audio contains speech bydetermining a beginning and an end of an utterance. The part of incomingaudio corresponding to speech is provided to a speech recognizer toobtain a speech recognition result or a transcript of the audio. Userperceived latency of speech recognition is the time from when the userstops speaking until the speech recognition result or transcript isoutput, often output for display on a screen of a user device. Therecent advancements made to run speech recognition models on-device haveallowed for real-time speech recognition results (e.g., streamingtranscription) to display as the user speaks and before endpointingdetermines the end of the utterance.

Yet, while the user perceived latency of the actual speech recognitionhas been improved by processing speech on-device, a user perceivedlatency of fulfillment of user queries/commands still exists since anendpoint identifying the end of the utterance has to be made before theactual fulfillment can take place. For example, a command for “Play rockmusic playlist on YouTube Music” spoken by a user to a digital assistantinterface executing on the user's phone requires performance of multiplesub-actions in order to fulfill the complete action of streaming theuser's rock music playlist from the YouTube Music service for audibleoutput through an acoustic speaker. These sub-actions include launchingthe YouTube Music application on the user's phone, accessing a searchbox in the YouTube Music to input a query for the rock music playlist orswitching to a playlists tab in the YouTube Music to select the rockmusic playlist, and finally executing audible playback of the rock musicplaylist from the YouTube Music application. In this example, the systemhas to wait for the end pointer to endpoint the end of the utterance andobtain the final speech recognition result before initiating the firstsub-action that needs to be fulfilled, e.g., launching the YouTube Musicapplication.

Implementations herein are directed to generating intermediate speechrecognition results from a user utterance commanding performance of anaction and using the intermediate speech recognition results to performmultiple sub-actions related to the final action while the user is stillspeaking the utterance. The leveraging of available intermediate speechrecognition results to identify and to perform sub-actions related to afinal action before the user finishes speaking drastically reduces userperceived latency since waiting for a final endpoint is not requiredbefore initiating performance of a first sub-action in a sequence ofsub-actions that need to be performed in order to fulfill the finalaction.

Yet, intermediate recognition results of “Play”, “rock music”, and“playlist” occur before the final endpoint and are available to identifypossible sub-actions related to the query. For instance, theintermediate recognition result of “Play” can indicate that the userwants to play media content, which could include video content such astelevision programming, movies, or video clips, or audio content fromone of multiple music applications on the user device. Next, theintermediate recognition result of “rock music” provides context thatnarrows the type of media content the user wants to play to audiocontent. Now, the possible available sub-actions are opening either astreaming radio application, a Spotify application, or a YouTubeapplication on the user's phone that all can potentially output rockmusic. For instance, the streaming radio, Spotify, and YouTubeapplications all include their own rock music channels, while theSpotify and YouTube applications each include respective rock musicplaylists for the user. Once the intermediate recognition result of“playlist” becomes available, the available possible sub-actions are nowmore specific to indicate that a rock music playlist needs to beaccessed on the Spotify application or the YouTube application.

Referring to FIG. 1 , in some implementations, the speech environment100 includes a user 10 speaking an utterance 20 directed toward avoice-enabled device 110 (also referred to as a device 110 or a userdevice 110) executing a digital assistant interface 200. Here, theutterance 20 spoken by the user 10 may be captured by the device 110 instreaming audio 15 and may correspond to a query to perform an action24, and more specifically, a query 22 for the digital assistantinterface 200 to perform an action 24. The user 10 may prefix the query22 with a hotword 21 (e.g., invocation phrase) to trigger the device 110from a sleep or hibernation state when the hotword 21 is detected in thestreaming audio 15 by a hotword detector running on the device 110 whilein the sleep or hibernation state. The action 24 may also be referred toas an operation or task. In this sense, the user 10 may haveconversational interactions with the digital assistant interface 200executing on the voice-enabled device 110 to perform computingactivities or to find answers to questions.

The device 110 may correspond to any computing device associated withthe user 10 and capable of capturing audio from the environment 100.Some examples of user devices 110 include, but are not limited to,mobile devices (e.g., mobile phones, tablets, laptops, e-book readers,etc.), computers, wearable devices (e.g., smart watches), music players,casting devices, smart appliances (e.g., smart televisions) and internetof things (IoT) devices, remote controls, smart speakers, etc. Thedevice 110 includes data processing hardware 112 d and memory hardware112 m in communication with the data processing hardware 112 d andstoring instructions, that when executed by the data processing hardware112 d, cause the data processing hardware 112 d to perform one or moreoperations related to speech processing.

The device 110 further includes an audio subsystem with an audiocapturing device (e.g., an array of one or more microphones) 114 forcapturing and converting audio within the speech environment 100 intoelectrical signals (e.g., audio data 14 (FIGS. 2A-2E). While the device110 implements the audio capturing device 114 (also referred togenerally as a microphone 114) in the example shown, the audio capturingdevice 114 may not physically reside on the device 110, but be incommunication with the audio subsystem (e.g., peripherals of the device110). For example, the device 110 may correspond to a vehicleinfotainment system that leverages an array of microphones positionedthroughout the vehicle.

The device 110 may also include a display 116 to display graphical userinterface (GUI) elements (e.g., windows, screens, icons, menus, etc.)and/or graphical content. For example, the device 110 may load or launchapplications 118, 118 a-n that generate GUI elements or other graphicalcontent for the display 116. Moreover, the elements generated in thedisplay 116 may be selectable by the user 10 and also serve to providesome form of visual feedback to processing activities and/or operationsoccurring on the device 110. Furthermore, since the device 110 is avoice-enabled device 110, the user 10 may interact with elementsgenerated on the display 116 using various voice commands. For instance,the display 116 may depict a menu of options for a particularapplication 118 and the user 10 may use the interface 200 to select anoption through speech.

In some implementations, the device 110 communicates via a network 120with a remote system 130. The remote system 130 may include remoteresources 132, such as remote data processing hardware 134 (e.g., remoteservers or CPUs) and/or remote memory hardware 136 (e.g., remotedatabases or other storage hardware). The device 110 may utilize theremote resources 132 to perform various functionality related to speechprocessing. For instance, some portion of the assistant interface 200may reside on the remote system 130. In one example, a speech recognizer210 executes locally on the device 110 to provide on-device automatedspeech recognition (ASR). In another example, the speech recognizer 210resides on the remote system 130 to provide server-side ASR for theassistant interface 200. In yet another example, functionality of thespeech recognizer 210 is split across the device 110 and the server 130(e.g., the device 110 and the server 130 perform parallel ASRprocessing).

The speech-enabled interface 200 may field the query/command 22 conveyedin the spoken utterance 20 captured in streaming audio 15 by the device110. The speech-enabled interface 200 (also referred to as the interface200 or the assistant interface 200) generally facilitates receivingaudio data 14 corresponding to an utterance 20 captured by the audiocapturing device 114, coordinating speech processing on the audio data14, performing semantic interpretation on the audio data 14 to identifya query 22 to perform an action 24, and performing a sequence ofsub-actions 26, 26 a-n in order to fulfill the action 24. The interface200 may execute on the data processing hardware 112 d of the device 110.When the microphone 114 of the device 110 captures an utterance 20 instreaming audio 15 and converts the audio into audio data 14, the audiodata 14 corresponding to the utterance 20 is relayed to the interface200 such that the interface 200 can perform speech recognition andsemantic interpretation to identify and ultimately fulfill a query 22conveyed in the spoken utterance 20. Although traditionally, fulfillmentsystems have had to wait for an endpointer (e.g., part of the speechrecognizer or a separate component) to endpoint the end of an utterance20 to obtain the final speech recognition result before initiatingperformance of an action 24 specified by the query/command 22, here, theinterface 200 fulfills sub-actions 26 associated with the final action24 as partial speech recognition results (e.g., partial hypotheses) aregenerated and before the user 10 is finished speaking the query 22 inthe utterance 20. By leveraging partial speech recognition results, theinterface 200 may provide the user 10 with real-time or near real-timefeedback as to sub-actions 26 the interface 200 is performing in orderto fulfill an action 24 specified by the query 22 once the utterance 20is endpointed. In other words, the interface 200 strives to performsub-actions 26 interpreted from intermediate speech recognition resultsfor the audio data 14 while the user 10 is actively speaking theutterance 20.

In some implementations, to perform a sequence of sub-actions 26 inorder to fulfill an action 24, the interface 200 interfaces with aplurality of applications 118, 118 a-n on the device 110 or accessibleto the device 110. An application 118 generally refers to anyapplication 118 that is configured to run on the device 110. Some typesof applications 118 include media applications (e.g., video streamingapplications, audio streaming applications, media player applications,media gallery applications, etc.), word processing applications,navigation applications, social media applications, communicationapplications (e.g., messaging applications, email applications, etc.),financial applications, organizational applications (e.g., address bookapplications), retail applications, entertainment applications (e.g.,news applications, weather applications, sport applications), castingapplications, etc. The interface 200 may be integrated with theseapplications 118 to enable the user 10 to control applications on thedevice 110 using his or her voice. For example, the interface 200 is anapplication programming interface (API) or any other type of program orapplication configured to execute the functionality of the interface200.

With continued reference to FIG. 1 , the user 10 speaks an utterance 20that states “hey computer, show photos from my holiday trip to Greecewith my parents.” Here, the utterance 20 is a voice query/command 22asking the interface 200 to perform the action 24 of displaying aparticular set of the user's photos. While the interface 200 receivesaudio data 14 corresponding to this utterance 20, portions of theutterance 20 are processed as they are received at the interface 200 inorder to generate intermediate speech results 212 (FIGS. 2A-2E). Theintermediate speech results 212 are interpreted by the interface 200 toidentify sub-actions 26 associated with a candidate action 24 while theuser 10 is speaking the utterance 20. For instance, the complete action24 specified by the query 22 once the utterance 20 is complete includesa sequence of four sub-actions 26, 26 a-d that may be performed as theuser 10 speaks the utterance 20 rather than waiting for the user 10 tofinish speaking the utterance 20.

Referring to FIGS. 2A-2H, the assistant interface 200 generally includesa speech recognizer 210, an interpreter 220, and an executor 230. Thespeech recognizer 210 receives audio data 14 as an input and processesthe audio data 14 to generate a speech recognition result 212. As thespeech recognizer 210 performs speech recognition on the audio data 14,the speech recognizer 210 may generate intermediate speech results 212.Here, an intermediate speech recognition result 212 refers to a resultthat is generated over some subset of the audio data 14 (e.g., a portionof the audio data) instead of the entirety of the audio data 14.Generally speaking, speech recognition may occur for an entire phrase(e.g., an entire utterance) or some subset of a phrase, such ascharacters, wordpieces, and/or words. Stated differently, eachrecognition result in a sequence of speech recognition results (e.g., asequence of intermediate speech recognition results 212) may correspondto a character, wordpiece, and/or word combined to form a finalrecognition result (e.g., transcription) of the utterance 20. Forinstance, when speech recognition occurs over an audio stream, a speechrecognition model performing speech recognition may generate an output(i.e., a result or hypothesis) at each time step in a streaming fashion.Here, the frame-by-frame outputs may correspond to intermediate speechrecognition results that may be combined to form one or more sequencesof speech recognition results 212 representative of respective portionsof the audio data 14. In some configurations, a speech recognitionresult corresponds to the top-N hypotheses at any given time step suchthat multiple hypotheses may be processed at once for generating allpossible partial query interpretations. Since the audio data 14 maycorrespond to an utterance 20 spoken by the user 10, a portion of theaudio data 14 may correspond to a portion of the utterance 20. Forexample, when the audio data 14 corresponds to the utterance 20 of “showphotos from my holiday trip to Greece with my parents,” a portion of theaudio data 14 may be a sequence of characters forming one or more words.As shown FIGS. 2B-2D, a first sequence of intermediate speechrecognition results 212 a for a first portion 14, 14 a of the audio data14 includes the phrase “show photos.” A second sequence of intermediatespeech recognition results 212 b for a second portion 14, 14 b of theaudio data 14 includes the phrase “from my holiday trip to Greece.” Athird sequence of intermediate speech recognition results 212 c for afourth portion 14, 14 c of the audio data 14 includes the phrase “withmy parents.”

The interpreter 220 receives one or more sequences of intermediatespeech recognition results 212 and performs partial query interpretationon the one or more intermediate speech recognition results 212. Forinstance, the interpreter 220 performs semantic interpretation (e.g.,grammar interpretation) on a sequence of intermediate speech recognitionresults 212 to understand a portion of the utterance 20 and its contextto identify any candidate sub-actions 26 that may be associated with afinal action 24 to be specified once the query 22 is revealed when theuser 10 finished speaking the utterance 20. Here, because theinterpreter 220 is interpreting a sequence of intermediate speechrecognition results 212 that corresponds to only a portion of the query22, the interpreter 220 is able to derive the context of a sub-action 26from the sequence of intermediate speech recognition results 212corresponding to a portion of the utterance 20. Accordingly, the resultof the partial query interpretation performed on a sequence ofintermediate speech results 212 includes an interpretation 222 that maycharacterize a sub-action 26 that the query/command 22 will involve inorder to fulfill a final action 24 that remains unknown until the useris finished speaking the utterance 20. When the interpreter 220 performspartial query interpretation, the interpretation 222 may suffer fromsome missing information due to the inherent fact that theinterpretation is unable to contextualize the entirety of the utterance20. Stated differently, the sub-actions 26 characterized byinterpretations 222 may become increasingly specific as the number ofsequences of intermediate speech recognition results 212 generated fromthe audio data 14 increases. For this reason, the interpreter 220 mayform as complete of an interpretation 222 as possible by deriving amissing intent from the available information from a sequence ofintermediate speech recognition results 212. For example, theinterpreter 220 may perform partial query interpretation on an initialsequence of intermediate speech recognition results 212 to identify aparticular application type needed to perform an action 24, but fails tospecify a slot value associated with naming a specific application 118for use in fulfilling the action 24 since the interpreter 220 cannotconfidently identify the specific application 118 from the initialsequence of intermediate speech recognition results 212. In thisexample, the executor 230 may be configured to perform a firstsub-action 26 by launching a default application 118 associated with theparticular application type since the slot value associated with namingthe specific application 118 is not specified (e.g., empty).

Referring to the example shown in FIG. 2B, when the interpreter 220generates a first interpretation 222, 222 a for the phrase “show mephotos” indicated by the first sequence of intermediate speechrecognition results 212, 212 a, the interpreter 220 is clearly lacking,at the time of interpretation, the context of which particular photos toshow. Because of this limited context, the interpreter 220 generates amore general interpretation 222, 222 a that indicates that the user 10is likely issuing a query 22 that will likely require launching a photoapplication 118 on the device 110. This means that the interpreter 220may infer a particular application type (e.g., photo applications) fromthe first sequence of intermediate speech recognition results 212 a,“show me photos,” even though a name of a specific photo applicationcannot be derived from the partial query interpretation performed on thefirst sequence of intermediate speech recognition results 212 a.Moreover, as the user 10 continues to speak the utterance 20, the speechrecognizer 210 generates additional sequences of intermediate speechrecognition results 212, 212 b-d and the interpreter 220 performspartial query interpretation on each of the sequences of intermediatespeech recognition results 212 to generate respective interpretations222, 222 b-d that provide additional context to the query 22. In otherwords, the interpreter 220 performs partial query interpretation on eachsequence of intermediate speech recognition results 212 while the user10 is speaking to identify and construct a sequence of sub-actions 26that need to be performed in order to fulfill an action 24. The executor230 may execute some or all of these sub-actions 26 before the user isfinished speaking and then ultimately complete fulfillment of the action24 responsive to receiving an end of speech condition when user 10 isfinished speaking the utterance 20. At the time the end of speechcondition is received, the executor 230 may have already performed eachsub-action 26 in the sequence of sub-actions 26 that are needed in orderto fulfill the query 22, and thus, fulfilled the action 24 by the timethe end of speech condition is received.

In FIG. 2C, the interpreter 220 performs partial query interpretation onthe second sequence of intermediate speech recognition results 212, 212b of “from my holiday trip to Greece” to generate a secondinterpretation 222, 222 b that identifies a search filter (e.g., searchquery) on a variable of location with respect the user 10 within thephoto application 118. Based on the second interpretation 222 b, theexecutor 230 may perform a second sub-action 22, 26 c by filteringphotos in the local photo gallery application on the variable oflocation). For instance, FIG. 2C illustrates the user's albums arefiltered on the variable of location.

In some examples, a subsequent interpretation 222 may void a priorinterpretation 222. For instance, if the first interpretation 222, 222 aled to the executor 230 launching two plausible photo applications 118in parallel where one was a local photo gallery application and theother was a third-party photo gallery accessible via a browserapplication, the second interpretation 222, 222 b that identifies thatthe filter of time relates specifically to the user 10 (e.g., based onthe word “my” preceding the word “holiday) would void/rollback theaction of opening the third-party photo gallery because that gallerywill not include any user-specific photos.

Referring to FIG. 2D, the interpreter 220 performs partial queryinterpretation on the third sequence of intermediate speech recognitionresults 212, 212 c of “with my parents” to generate a thirdinterpretation 222, 222 c that identifies another search filter (e.g.,search query) for the presence of the user's parents within the photos.This means that the third interpretation 222 c will result in theexecutor 230 performing a third sub-action 26, 26 c of further filteringthe photos in the local photo gallery application by subject matter(i.e., the subject matter of the user's parents).

In some configurations, the interpreter 220 uses an interpretation modelthat generates a confidence level for a given interpretation 222. Insome implementations, the interpreter 220 generates multiple possibleinterpretations 222 for the same sequence of intermediate speechrecognition results 212 and each possible interpretation 222 may have arespective confidence level. Furthermore, the speech recognizer 210 maygenerate multiple different candidate sequences of intermediate speechrecognition results 212 for a same portion of audio data 14 and theinterpreter 220 may generate one or more possible interpretations 222for each candidate sequence of intermediate speech recognition results222. Yet in some approaches, the interface 200 may only want to pursue alimited number of interpretations 222 (e.g., one interpretation 222) orinterpretations 222 that indicate a confidence level above someinterpretation confidence threshold. Here, when the interpreter 220generates multiple possible interpretations 222 for a given sequence ofintermediate speech recognition results 212 and with confidence levelssatisfying an interpretation confidence threshold, the executor 230 mayprocess respective sub-actions 26 characterized by the possibleinterpretations 222 in parallel. With the sub-actions 26 processing inparallel, the interface 200 may graphically display each parallel trackon the display 116 and enable the user 10 to select a particular track,or even modify his or her utterance 20 to change the behavior of theinterpreter 220 and/or executor 230.

Proceeding with the example of FIGS. 2B-2D, when the executor 230receives the first interpretation 222 a that identifies photoapplications as a particular application type needed to perform anaction 24, the executor 230 performs a first sub-action 26 a bylaunching a first photo application 118. Here, while the partial queryinterpretation performed on the first sequence of intermediate ASRresults 212 a identifies the particular application type (e.g., photoapplications), the partial query interpretation fails to specify a slotvalue associated with naming a specific photo application for use infulfilling the action 24. Accordingly, the first photo application 118corresponds to a default photo application that the user 10 may havepreviously specified to use as default (e.g., for photos) or one theuser 10 uses most frequently. As shown in FIG. 2B, the display 116 maydepict the launching of the first photo application 118 by showing agraphical icon representing the first photo application 118 beingselected and proceeding to display a graphical user interface (GUI) 240for the first photo application 118.

When, as shown in FIG. 2C, the executor 230 receives the secondinterpretation 222 b that identifies the search filter relating tophotos with the user during holiday time in Greece, the executor 230performs a second sub-action 26 b by instructing the first photoapplication 118 launched on the device 110 to perform a search query forphotos of the user during holiday time in Greece. For instance,comparing FIGS. 2B and 2C, originally the application 118, whenlaunched, displayed two photo albums sorted by location. One albumdepicting London photos and the other album depicting photos fromGreece. Here, the executor 230 may instruct the application to inputtext characterizing the search query (e.g., text saying “Greece” as asearch term 242 _(ST)) into a search field 242 of the GUI 240 for thefirst photo application 118 displayed on the display 116. By executingthis search query, FIG. 2C depicts the nine photo thumbnails from thealbum “Greece.” Following the second sub-action 26 b, the executor 230receives the third interpretation 222 c that identifies the searchfilter relating to photos with the user 10 in Greece that includes theuser's parents. The executor 230 performs the third sub-action 26 c bysorting the photos in the “Greece” album by people. Here, thissearch/filter selects four of the nine photos that resulted from theprevious sub-step 26 b and displays these four photos within the firstphoto application 118. As illustrated by FIG. 2D, the GUI 240 for thefirst photo application 118 includes two different methods havingdifferent GUI elements for searching or filtering content within thefirst photo application 118. In this example, the user 10 may use afirst GUI element corresponding to a search field 242 to search thecontent of the application 118 (e.g., to search photos within the firstphoto application 118). The application 118 also includes GUI elementcorresponding to a menu of sorting/featuring options. Specific to thisexample, the menu 244 depicts a list of photo tags that allow the user10 to sort his or her photos by tags that have been associated withphotos of the first photo application 118. A tag generally refers to anidentifier that may be shared across one or more content elements (e.g.,photos). In this example, the executor 230 may select the option of“parents” within the menu 244 to instruct the executor 230 to filter theuser's photos of Greece by whether the user's parents are present.

In some configurations, when the utterance 20 is complete or endpointed,the executor 230 performs complete fulfillment of the action 24. Here,the utterance 20 may be endpointed when the speech recognizer 210detects some designated minimum duration of time of non-speech in theaudio data 14. The executor 230 may perform a highest confidence action24 based on the full speech recognition result for the entirety of theaudio data 14 (or utterance 20). In these configurations, the executor230 may roll back (rescind or terminate) previous sub-actions 26 thatoccurred prior to the fulfillment of the entire action 24. For instance,the executor 230 rolls back one or more sub-actions 26 that areinconsistent with the entire action 24. Generally speaking, thesub-action execution process aims to be in a final state that matchesthe final state of the execution process of the full action 24. However,depending on the query/command 22, this may not always be the case.Hence, roll back(s) allow the interface 200 to flexibly accommodate fordifferent scenarios.

In some implementations, the executor 230 may roll back previoussub-actions 26 based not only on the entire action 24, but on othersub-actions 26 prior to the utterance 20 being endpointed. Toillustrate, as described above, the interpreter's first interpretation222 may have led to two plausible photo-related types of applications118 where one was a local photo gallery application and the other was athird-party photo gallery in a browser application. Yet laterinterpretations 222 of sub-actions 26 confirmed that the application 118could not have been the third-party photo gallery application because itwould not contain photos of the user 10. In this respect, the executor230 would roll back the first sub-action 26 for the launching of thethird-party photo gallery in favor of the local photo galleryapplication. If the executor 230 actually launched the third-party photogallery, but did not also launch the local photo gallery application,the executor 230, based on the later interpretations 222, may roll backthe first sub-action 26 for the launching of the third-party photogallery by ceasing execution of the third-party photo gallery andinstead re-perform the first sub-action 26 by launching the local photogallery application.

Additionally or alternatively, there may be certain sub-actions 26 thatthe interface 200 is not able to roll back, or that rolling back thesub-action 26 would negatively impact the user's experience. Forexample, when the sub-action 26 is to purchase an item on a retailapplication 118, the interface 200 may not be able to roll back such apurchase or do so without user intervention. Here, a sub-action 26 thatthe interface 200 is not able to roll back may be referred to as anirreversible sub-action 26. In these irreversible sub-actions 26, theexecutor 230 may prompt the user 10 for authorization or actionconfirmation while executing an irreversible sub-action 26. Anotherapproach to irreversible sub-actions 26 is to identify an irreversiblesub-action 26 and, when a sub-action 26 is identified as irreversible,the executor 230 waits to perform complete fulfillment of the action 24.That is, it may be safer to have the entire context of a full speechrecognition result interpreted rather than a partial speech recognitionresult 212. In yet another approach, before the utterance 20 isendpointed, but after the executor 230 launches an application 118, theexecutor 230 may determine a rollback feasibility score for a sub-action26. Here, the rollback feasibility score indicates a likelihood that theuser's experience will be degraded or detrimentally impacted if theexecutor 230 rolls back the sub-action 26. When the rollback feasibilityscore satisfies a rollback feasibility threshold, the executor 230 mayproceed to execute the sub-action 26. On the other hand, when therollback feasibility score fails to satisfy the rollback feasibilitythreshold, the executor 230 may roll back the sub-action 26 or delay theroll back of the sub-action 26 to determine whether the completefulfillment of the action 24 indicates that the sub-action 26 should berolled back or not, and rolling it back accordingly.

Because the user 10 may see results of sub-actions 26 being performed bythe interface 200 while the user 10 is speaking, the user 10 may, insome circumstances, endpoint an utterance manually before the userfinishes speaking the complete utterance. Namely, when the user 10 isseeking a particular result from the device 110, if the particularresult is displayed for the user 10 prior to completing the utterance20, the user 10 may abandon the completion of the utterance 20 since thepurpose has already been achieved. For instance, using the example ofFIGS. 2B-2D, if all of the user's photos in the Greece album includedthe user's parents, the second sub-action 26 b and the third sub-action26 c would produce the same results. In this scenario, the user 10 mayrecognize that the device 110 has successfully displayed his or herphotos with his or her parents from the Greece trip. Based on thisrecognition, the user 10 may endpoint the utterance 20 early (i.e.,before stating the portion, “with my parents”).

FIGS. 2E-2H show an example interface 200 while the executor 230 isexecuting multiple tasks in parallel based on an utterance 20 spoken bythe user 10. In this example, the user 10 speaks an utterance 20directed toward the device 110 that includes a query 22 “play rock musicplaylist on Google Play music.” When the user 10 speaks the utterance20, the interface receives the audio data 14 corresponding to theutterance 20 and relays the audio data 14 to the speech recognizer 210.For a first portion 14, 14 a of the audio data 14 including the word“play,” the speech recognizer 210 provides a first sequence ofintermediate speech recognition results 212, 212 a as shown in FIG. 2E.The interpreter 220 receives and performs partial query interpretationon the first sequence of speech recognition results 212 a to obtain afirst interpretation 222, 222 a. Based on the word “play,” the firstinterpretation 222 a indicates that the audio data 14 includes amedia-related command to execute a media type of application 118 and theexecutor 230 performs a first set of four sub-actions 26, 26 a-d thatlaunch four different applications 118, 118 a-d on the device 110. Thefirst application 118 a includes a first music streaming application,Google Play. The second application 118 b includes a second musicstreaming application, Spotify. The third application 118 c includes avideo streaming application, YouTube, and the fourth application 118 dincludes a local generic video player application called “video player.”These four applications 118 a-d may be launched to execute in parallelon the device 110 as the sequence of sub-actions 26 continues and thequery/command spoken in the utterance 20 becomes more specific. When theexecutor 230 launches these applications 118 a-d, these applications 118a-d may appear on the device 110 in one or more windows (e.g., a cascadeof windows or a panel of windows). Because the first portion of theutterance 20 lacks a clear and identifiable application 118 to “play”something, the interpreter 220 generates multiple candidate sub-actions26 a-d, which may include all of the media type of applications 118rather than a specific application 118 at this point when the interfaceis beginning to perform the action 24 corresponding the command 22 ofthe utterance 20. Furthermore, since the execution process for eachsub-action 26 may be displayed on the display 116 of the device 110 forthe user 10, the user 10 (e.g., while speaking the utterance 20), mayengage with the display 116 to further facilitate or modify thesub-actions 26. For instance, with four applications 118 displayed forthe user 10, the user 10 may close one or more applications 118 byinteracting with a GUI element of a GUI 240 for the application 218 thathas the functionality to terminate a selected application 118. Incontrast, the user 10 may additionally or alternatively select whichapplications 118 to keep rather than to terminate. In yet anotherexample, the user 10 may select to keep an application 118 and also toterminate an application 118.

As the user 10 continues to speak the utterance 20, FIG. 2F depicts thespeech recognizer 210 performing speech recognition over a secondportion 14 b of the audio data 14 relating to the words “rock music” togenerate a second sequence of intermediate speech recognition results212, 212 b. The interpreter 220 interprets that the second sequence ofintermediate speech recognition results 212 b to obtain two differentinterpretations 222, 222 b-c. A first interpretation 222 b where “rockmusic” refers to a genre of music (i.e., the genre of rock music) or asecond interpretation 222 where “rock music” refers to a playlist (e.g.,playlist title) for the user 10. Since both of these interpretations 222b-c may have a confidence level that satisfies the confidence threshold,the executor 230 may generate parallel processes to execute bothinterpretations 222 b-c on each application 118 launched and open fromthe first set of sub-actions 26 a-d.

The executor 230 may also use the second interpretation 222 b and thethird interpretation 222 c as a validity check to determine if any ofthe sub-actions 26 in the first set of sub-actions 26 a-d should berolled back (e.g., terminated). Since two of the applications 118 c-dwere applications for video (e.g., streaming video or a local videoplayer), performing the validity check against the second interpretation222 b and the third interpretation 222 c results in the executor 230terminating the video-based applications 118 c-d that were launchedbased on a third and a fourth sub-action 26, 26 c-d. With this being thecase, executing both interpretations 222 b-c only has to occur at thefirst application 118 a and the second application 118 b. This executionprocess therefore forms a second set of sub-actions 26 e-h where fourparallel action sequences are occurring as shown in FIG. 2F.

In FIG. 2G, while the user 10 continues to speak the utterance 20, thespeech recognizer 210 performs speech recognition over a third portion14 c of the audio data 14 relating to the word “playlist” to generate athird sequence of intermediate speech recognition results 212, 212 cthat the interpreter 220 performs partial query interpretation on todetermine a fourth interpretation 222, 222 d. In this step of theprocessing, the interpreter 220 may be able to recognize that thissequence of intermediate speech recognition results 212, 212 c clarifiessome of the ambiguity of the second and third interpretations 222 b-c.Namely, that “rock music” should not have been interpreted as a genresearch for music, but rather referred to a playlist called “rock music.”Based on these additional context clues, the executor 230 has terminatedany of the processes that were executing on the premise that “rockmusic” may correspond to a genre. After the termination of theseprocesses, this leaves two remaining action sequences, one for the firststreaming music application 118 of Google Play music and one for thesecond streaming music application 118 of Spotify. At this point in theaction sequence, each application 118 has been launched and has searchedfor the playlist “rock music” according to the sub-actions 26 i-jcorresponding to the fourth interaction 222 d.

In FIG. 2G, the user 10 speaks the final portion of the utterance 20.While the user 10 speaks the final portion of the utterance 20 andbefore an endpoint 214 (FIG. 2H) (or end of speech condition) of theutterance 20, the speech recognizer 210 performs speech recognition overa fourth portion 14, 14 d of the audio data 14 relating to the words,“on Google Play music.” Here, the speech recognizer 210 generates afourth sequence of intermediate speech recognition results 212, 212 dfor the fourth portion 14 d of the audio data 14. The speech recognizer210 relays the fourth sequence of intermediate speech recognitionresults 212 d to the interpreter 220 and the interpreter 220 performs apartial query interpretation to generate a fifth interpretation 222, 222e that identifies the fourth sequence of intermediate speech recognitionresults 212 d as uniquely designating a particular application, GooglePlay music. In other words, although this specific application 118 wasdefined at the end of the utterance 20, this application 118 was alreadyexecuting while the user 10 was speaking because of the context of theutterance 20 preceding this portion of the words “on Google Play music.”Based on this fifth interpretation 222 e, the executor 230 ceases theexecution of the Spotify application 118. Additionally, when theendpoint 214 of the utterance 20 occurs, the interpreter 220 may thencompare the sub-action sequence to the action 24 to fulfill the fullspeech recognition result from the speech recognizer 210.

FIG. 3 is a flowchart of an example arrangement of operations forstreaming action fulfillment based on partial hypotheses. At operation302, the method 300 receives audio data 14 corresponding to an utterance20 spoken by a user 10 of a user device 110 where the utterance 20includes a query 22 to perform an action 24 where the query 22 requiresperformance of a sequence of sub-actions 26 in order to fulfill theaction 24. Operation 304 includes three sub-operations 304, 304 a-c thatoccur while the method 300 receives the audio data 14, but beforereceiving an end of speech condition 214. At operation 304 a, the method300 processes, using a speech recognizer 210, a first portion 14, 14 aof the received audio data 14 to generate a first sequence ofintermediate automated speech recognition (ASR) results 212. Atoperation 304 b, the method 300 performs partial query interpretation onthe first sequence of intermediate ASR results 212 to determine whetherthe first sequence of intermediate ASR results 212 identifies anapplication type needed to perform the action. At operation 304 c, whenthe first sequence of intermediate ASR results 212 identifies aparticular application type, performing a first sub-action 26 in thesequence of sub-actions 26, 26 a-n by launching a first application 118to execute on the user device 110 where the first application 118 isassociated with the particular application type. At operation 306, themethod 300 includes, in response to receiving an end of speech condition214, fulfilling performance of the action 24.

FIG. 4 is a schematic view of an example computing device 400 that maybe used to implement the systems (e.g., the interface 200) and methods(e.g., the method 300) described in this document. The computing device400 is intended to represent various forms of digital computers, such aslaptops, desktops, workstations, personal digital assistants, servers,blade servers, mainframes, and other appropriate computers. Thecomponents shown here, their connections and relationships, and theirfunctions, are meant to be exemplary only, and are not meant to limitimplementations of the inventions described and/or claimed in thisdocument.

The computing device 400 includes a processor 410, memory 420, a storagedevice 430, a high-speed interface/controller 440 connecting to thememory 420 and high-speed expansion ports 450, and a low speedinterface/controller 460 connecting to a low speed bus 470 and a storagedevice 430. Each of the components 410, 420, 430, 440, 450, and 460, areinterconnected using various busses, and may be mounted on a commonmotherboard or in other manners as appropriate. The processor 410 canprocess instructions for execution within the computing device 400,including instructions stored in the memory 420 or on the storage device430 to display graphical information for a graphical user interface(GUI) on an external input/output device, such as display 480 coupled tohigh speed interface 440. In other implementations, multiple processorsand/or multiple buses may be used, as appropriate, along with multiplememories and types of memory. Also, multiple computing devices 400 maybe connected, with each device providing portions of the necessaryoperations (e.g., as a server bank, a group of blade servers, or amulti-processor system).

The memory 420 stores information non-transitorily within the computingdevice 400. The memory 420 may be a computer-readable medium, a volatilememory unit(s), or non-volatile memory unit(s). The non-transitorymemory 420 may be physical devices used to store programs (e.g.,sequences of instructions) or data (e.g., program state information) ona temporary or permanent basis for use by the computing device 400.Examples of non-volatile memory include, but are not limited to, flashmemory and read-only memory (ROM)/programmable read-only memory(PROM)/erasable programmable read-only memory (EPROM)/electronicallyerasable programmable read-only memory (EEPROM) (e.g., typically usedfor firmware, such as boot programs). Examples of volatile memoryinclude, but are not limited to, random access memory (RAM), dynamicrandom access memory (DRAM), static random access memory (SRAM), phasechange memory (PCM) as well as disks or tapes.

The storage device 430 is capable of providing mass storage for thecomputing device 400. In some implementations, the storage device 430 isa computer-readable medium. In various different implementations, thestorage device 430 may be a floppy disk device, a hard disk device, anoptical disk device, or a tape device, a flash memory or other similarsolid state memory device, or an array of devices, including devices ina storage area network or other configurations. In additionalimplementations, a computer program product is tangibly embodied in aninformation carrier. The computer program product contains instructionsthat, when executed, perform one or more methods, such as thosedescribed above. The information carrier is a computer- ormachine-readable medium, such as the memory 420, the storage device 430,or memory on processor 410.

The high speed controller 440 manages bandwidth-intensive operations forthe computing device 400, while the low speed controller 460 manageslower bandwidth-intensive operations. Such allocation of duties isexemplary only. In some implementations, the high-speed controller 440is coupled to the memory 420, the display 480 (e.g., through a graphicsprocessor or accelerator), and to the high-speed expansion ports 450,which may accept various expansion cards (not shown). In someimplementations, the low-speed controller 460 is coupled to the storagedevice 430 and a low-speed expansion port 490. The low-speed expansionport 490, which may include various communication ports (e.g., USB,Bluetooth, Ethernet, wireless Ethernet), may be coupled to one or moreinput/output devices, such as a keyboard, a pointing device, a scanner,or a networking device such as a switch or router, e.g., through anetwork adapter.

The computing device 400 may be implemented in a number of differentforms, as shown in the figure. For example, it may be implemented as astandard server 400 a or multiple times in a group of such servers 400a, as a laptop computer 400 b, or as part of a rack server system 400 c.

Various implementations of the systems and techniques described hereincan be realized in digital electronic and/or optical circuitry,integrated circuitry, specially designed ASICs (application specificintegrated circuits), computer hardware, firmware, software, and/orcombinations thereof. These various implementations can includeimplementation in one or more computer programs that are executableand/or interpretable on a programmable system including at least oneprogrammable processor, which may be special or general purpose, coupledto receive data and instructions from, and to transmit data andinstructions to, a storage system, at least one input device, and atleast one output device.

These computer programs (also known as programs, software, softwareapplications or code) include machine instructions for a programmableprocessor, and can be implemented in a high-level procedural and/orobject-oriented programming language, and/or in assembly/machinelanguage. As used herein, the terms “machine-readable medium” and“computer-readable medium” refer to any computer program product,non-transitory computer readable medium, apparatus and/or device (e.g.,magnetic discs, optical disks, memory, Programmable Logic Devices(PLDs)) used to provide machine instructions and/or data to aprogrammable processor, including a machine-readable medium thatreceives machine instructions as a machine-readable signal. The term“machine-readable signal” refers to any signal used to provide machineinstructions and/or data to a programmable processor.

The processes and logic flows described in this specification can beperformed by one or more programmable processors, also referred to asdata processing hardware, executing one or more computer programs toperform functions by operating on input data and generating output. Theprocesses and logic flows can also be performed by special purpose logiccircuitry, e.g., an FPGA (field programmable gate array) or an ASIC(application specific integrated circuit). Processors suitable for theexecution of a computer program include, by way of example, both generaland special purpose microprocessors, and any one or more processors ofany kind of digital computer. Generally, a processor will receiveinstructions and data from a read only memory or a random access memoryor both. The essential elements of a computer are a processor forperforming instructions and one or more memory devices for storinginstructions and data. Generally, a computer will also include, or beoperatively coupled to receive data from or transfer data to, or both,one or more mass storage devices for storing data, e.g., magnetic,magneto optical disks, or optical disks. However, a computer need nothave such devices. Computer readable media suitable for storing computerprogram instructions and data include all forms of non-volatile memory,media and memory devices, including by way of example semiconductormemory devices, e.g., EPROM, EEPROM, and flash memory devices; magneticdisks, e.g., internal hard disks or removable disks; magneto opticaldisks; and CD ROM and DVD-ROM disks. The processor and the memory can besupplemented by, or incorporated in, special purpose logic circuitry.

To provide for interaction with a user, one or more aspects of thedisclosure can be implemented on a computer having a display device,e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor, ortouch screen for displaying information to the user and optionally akeyboard and a pointing device, e.g., a mouse or a trackball, by whichthe user can provide input to the computer. Other kinds of devices canbe used to provide interaction with a user as well; for example,feedback provided to the user can be any form of sensory feedback, e.g.,visual feedback, auditory feedback, or tactile feedback; and input fromthe user can be received in any form, including acoustic, speech, ortactile input. In addition, a computer can interact with a user bysending documents to and receiving documents from a device that is usedby the user; for example, by sending web pages to a web browser on auser's client device in response to requests received from the webbrowser.

A number of implementations have been described. Nevertheless, it willbe understood that various modifications may be made without departingfrom the spirit and scope of the disclosure. Accordingly, otherimplementations are within the scope of the following claims.

What is claimed is:
 1. A computer-implemented method executed on dataprocessing hardware that causes the data processing hardware to performoperations comprising: receiving audio data corresponding to at least aportion of an utterance spoken by a user, the utterance comprising aquery to perform an action; processing, using a speech recognizer, theaudio data to generate a sequence of intermediate automated speechrecognition (ASR) results; based on the sequence of intermediate ASRresults, launching both a first application and a second application toexecute in parallel on a user device, the first application and thesecond application both associated with a particular application type;and in response to detecting an end of speech condition, fulfillingperformance of the action using one of the first application or thesecond application.
 2. The computer-implemented method of claim 1,wherein the operations further comprise, prior to detecting the end ofspeech condition: performing partial query interpretation on thesequence of intermediate ASR results to determine that the sequence ofintermediate ASR results identifies the particular application typeneeded to perform the action, wherein launching both the firstapplication and the second application based on determining that thesequence of intermediate ASR results identifies the particularapplication type needed to perform the action.
 3. Thecomputer-implemented method of claim 1, wherein the operations furthercomprise, prior to detecting the end of speech condition, displaying, ina graphical user interface of the user device, a first graphical elementand a second graphical element simultaneously, the first graphicalelement representing the first application executing on the user deviceand the second graphical element representing the second applicationexecuting on the user device.
 4. The computer-implemented method ofclaim 3, wherein the operations further comprise: receiving a user inputindication indicating selection of one of the first graphical element orthe second graphical element displayed in the graphical user interface;and in response to receiving the user input indication: maintainingexecution of the one of the first application or the second applicationthat is represented by the selected one of the first graphical elementor the second graphical element; and ceasing execution of the other oneof the first application or the second application that is representedby the one of the first graphical element or the second graphicalelement that was not selected by the user input indication.
 5. Thecomputer-implemented method of claim 4, wherein the operations furthercomprise detecting the end of speech condition in response to receivingthe user input indication.
 6. The computer-implemented method of claim4, wherein fulfilling performance of the action comprises fulfillingperformance of the action using the one of the first application or thesecond application that is represented by the selected one of the firstgraphical element or the second graphical element.
 7. Thecomputer-implemented method of claim 1, wherein: receiving the audiodata corresponding to at least the portion of the utterance comprisesreceiving the audio data corresponding to an initial portion of theutterance; and the operations further comprise, after launching thefirst application and the second application and prior to detecting theend of speech condition: receiving additional audio data correspondingto a remaining portion of the utterance; processing the second portionof the utterance to generate a final ASR result; and determining thatthe final ASR result identifies one of the first application or thesecond application to use for fulfilling the action.
 8. Thecomputer-implemented method of claim 7, wherein fulfilling performanceof the action comprises fulfilling performance of the action using theone of the first application or the second application that the finalASR result identifies to use for fulfilling the action.
 9. Thecomputer-implemented method of claim 1, wherein detecting the end ofspeech condition comprises detecting, using the speech recognizer, atleast a minimum duration of non-speech in the received audio data. 10.The computer-implemented method of claim 1, wherein the data processinghardware resides on the user device.
 11. A system comprising: dataprocessing hardware; and memory hardware in communication with the dataprocessing hardware, the memory hardware storing instructions that whenexecuted on the data processing hardware cause the data processinghardware to perform operations comprising: receiving audio datacorresponding to at least a portion of an utterance spoken by a user,the utterance comprising a query to perform an action; processing, usinga speech recognizer, the audio data to generate a sequence ofintermediate automated speech recognition (ASR) results; based on thesequence of intermediate ASR results, launching both a first applicationand a second application to execute in parallel on a user device, thefirst application and the second application both associated with aparticular application type; and in response to detecting an end ofspeech condition, fulfilling performance of the action using one of thefirst application or the second application.
 12. The system of claim 11,wherein the operations further comprise, prior to detecting the end ofspeech condition: performing partial query interpretation on thesequence of intermediate ASR results to determine that the sequence ofintermediate ASR results identifies the particular application typeneeded to perform the action, wherein launching both the firstapplication and the second application based on determining that thesequence of intermediate ASR results identifies the particularapplication type needed to perform the action.
 13. The system of claim11, wherein the operations further comprise, prior to detecting the endof speech condition, displaying, in a graphical user interface of theuser device, a first graphical element and a second graphical elementsimultaneously, the first graphical element representing the firstapplication executing on the user device and the second graphicalelement representing the second application executing on the userdevice.
 14. The system of claim 13, wherein the operations furthercomprise: receiving a user input indication indicating selection of oneof the first graphical element or the second graphical element displayedin the graphical user interface; and in response to receiving the userinput indication: maintaining execution of the one of the firstapplication or the second application that is represented by theselected one of the first graphical element or the second graphicalelement; and ceasing execution of the other one of the first applicationor the second application that is represented by the one of the firstgraphical element or the second graphical element that was not selectedby the user input indication.
 15. The system of claim 14, wherein theoperations further comprise detecting the end of speech condition inresponse to receiving the user input indication.
 16. The system of claim14, wherein fulfilling performance of the action comprises fulfillingperformance of the action using the one of the first application or thesecond application that is represented by the selected one of the firstgraphical element or the second graphical element.
 17. The system ofclaim 11, wherein: receiving the audio data corresponding to at leastthe portion of the utterance comprises receiving the audio datacorresponding to an initial portion of the utterance; and the operationsfurther comprise, after launching the first application and the secondapplication and prior to detecting the end of speech condition:receiving additional audio data corresponding to a remaining portion ofthe utterance; processing the second portion of the utterance togenerate a final ASR result; and determining that the final ASR resultidentifies one of the first application or the second application to usefor fulfilling the action.
 18. The system of claim 17, whereinfulfilling performance of the action comprises fulfilling performance ofthe action using the one of the first application or the secondapplication that the final ASR result identifies to use for fulfillingthe action.
 19. The system of claim 11, wherein detecting the end ofspeech condition comprises detecting, using the speech recognizer, atleast a minimum duration of non-speech in the received audio data. 20.The system of claim 11, wherein the data processing hardware resides onthe user device.